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Abstract: Online learning is increasingly ubiquitous in higher 
education. However, research regarding online teaching often focuses 
on the affordances of the online environment rather than on the 
quality of pedagogy. In this paper we consider how online learning 
could be enhanced using rich peclagogiccd models that are consistent 
with a wealth of existing knowledge on pedagogy for face-to-face 
settings. To do so, we apply an established framework, the Quality 
Teaching model, to explore pedagogy in the online environment and 
illustrate its potential benefits using a case study of 60 students in a 
tertiary mathematics teacher education program. We conclude that 
the use of an evidence-based pedagogical model can help guide online 
instructors in the development of high quality online courses. 


Introduction 

There is ample evidence that teaching quality is a key determinant of student learning 
outcomes during schooling (Darling-Hammond, 1999; Fullan, 2007; Kyriakides, 

Christoforou & Charalambous, 2013). However, in the higher education setting, particularly 
in the context of online learning, the evidence supporting a similar link is less robust. 
Moreover, given that technology acts as a mediator of the teaching and learning experience 
within online learning scenarios, methods commonly employed to determine the quality of 
teaching in face-to-face settings, particularly as it impacts on the learning experience of 
students, are often seen as unsuitable in this environment (Ginns & Ellis, 2007). 

Defining and evaluating the quality of teaching in online learning environments, 
which many have characterised as substantially different from traditional classrooms, is a 
central focus of recent educational research in online teaching (Garrison, 2011). Several 
examples of instructional principles for courses were developed early for the online medium 
with clear guidelines for staff-student interactions, encouraging cooperation and active 
learning, giving prompt feedback, and setting clear deadlines (Graham, Cagiltay, Lim, 
Craner, & Duffy, 2001). Whilst many of these earlier guidelines acknowledge pedagogy as 
important, they have tended to focus on the affordances of the specific online environment 
such as accessibility, communication, reliability of the interface, and bandwidth demand 
(Herrington, Herrington, Oliver, Stoney & Willis, 2001). More recent work centres on 
general pedagogical principles that would be applicable across any online delivery system 
(Kidney, Cummings, & Boehm, 2014; Margaryan, Bianco & Littlejohn, 2015). Some authors 
argue for more work in the online context in order to “inform learner outcomes, learner 
characteristics, course environment, and institutional factors related to delivery system 
variables in order to test learning theories and teaching models inherent in course design” 
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(Tallent-Runnels et al., 2006, p.93). Indeed, some argue specifically that the development of 
such models could benefit from drawing on the existing wealth of well-established research 
on classroom-based pedagogies (Haythomthwaite & Andrews, 2011). 

A major issue identified in early research on online learning when utilising 
comparisons with face-to-face teaching is that in non-classroom based environments the 
notion of a ‘lesson’ is substantially different (De Wever, Schellens, Valcke, & Van Keer, 
2006). Lessons in online and blended environments can be understood as ‘units of work’ 
delivered via a range of media including discussion forums, blogs and individual email 
communication with the teaching academic. Hence, it is primarily the interactions occurring 
through these media that can be studied and analysed. Typically, interactions are student- 
student, student-teacher, or student-content, and it has been found that there is an association 
between the frequency of interactions and increased student achievement (Bernard et al., 
2009; Tamin et al., 2011). Many different instruments and measures are available to analyse 
the content of online interactions, although concerns about the validity and reliability of some 
of them have been raised (De Wever et al., 2006). 

The purpose of the study reported in this paper is to explore the applicability of an 
evidence-based approach to evaluating pedagogy in classrooms for the review and refinement 
of teaching in online and blended environments for pre-service teachers. In so doing, we 
explore the potential of pedagogical frameworks for informing the improvement of teaching 
in the online environment. 

We undertake this exploration using the Quality Teaching (QT) model (NSW 
Department of Education and Training, 2003b), a conceptually and empirically robust model 
guiding developed to guide the quality of teaching in primary and secondary schools. Very 
minor adjustments to the wording of the coding instrument (NSW Department of Education 
and Training, 2003a) were made in order to apply the model to the specific features of 
‘lessons’ and ‘interactions’ in the online environment. We use a case study of two online 
courses to illustrate how a pedagogical model, in our case the QT model, can be used to 
analyse teaching in the online environment. The elements that constitute the model, described 
in the following section, provide a strong and accessible conceptual basis and set of 
principles for guiding course development and interactions online. These principles have 
been shown in face-to-face environments to be linked with improved teaching, improved 
outcomes for students and narrowing of equity gaps for students from traditionally under¬ 
represented groups and equity target groups (Amosa, Ladwig, Griffiths & Gore, 2007). 


The Quality Teaching Model 

Quality teaching and how we define and recognise it has been the object of research 
for many decades. In the most populated state in Australia, New South Wales (NSW), the 
Department of Education and Training commissioned the development of a model for 
teaching quality, comprehensive in scope, and applicable across all subject areas and grade 
levels, as a framework for teachers’ professional self-reflection and for school improvement 
practices. This Quality Teaching model (NSW Department of Education and Training, 
2003b) is now well established in NSW and Australian Capital Territory public schools and 
there is growing evidence of its efficacy for improving teaching and student learning 
outcomes (Gore, 2007; Gore, 2014; Gore & Bowe, 2015; Ladwig, Smith, Gore, Amosa, & 
Griffiths, 2007). 

The QT model is a refinement of the Productive Pedagogies model (Hayes, Lingard, 
& Mills, 2000) which in turn was an extension of Authentic Pedagogy (Newmann, 1996). It 
features teaching practices that have been linked to improved student outcomes and can be 
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characterised as representing three dimensions of pedagogy: pedagogy that promotes high 
levels of intellectual quality, pedagogy that promotes a quality learning environment, and 
pedagogy that develops and makes explicit to students the significance of their work (NSW 
Department of Education and Training, 2003b). Each of these three dimensions is elaborated 
through six elements as detailed in Figure 1. For brief explanations of each element please 
see Appendix 1. 

Dimensions 


Intellectual Quality 

Quality Learning 
Environment 

Significance 

Deep knowledge 

Explicit quality criteria 

Background Knowledge 

Deep understanding 

Engagement 

Cultural knowledge 

Problematic 

knowledge 

High expectations 

Knowledge integration 

Higher orderthinking 

Social support 

Inclusivity 

Metalanguage 

Students'self-regulation 

Connectedness 

Substantive 

communication 

Student direction 

Narrative 


Figure 1. Elements and dimensions of the Quality Teaching model (NSW Department of 
Education and Training, 2003a, p.10) 


Studies using the QT model are often designed around the observation of teachers and 
their interaction with students in the classroom (NSW Department of Education and Training, 
2003a; Gore et al., 2015). However, as we argued in the introduction, the ‘observation’ of 
virtual classroom practices requires a different approach, involving the observation of 
student-student, student-teacher, and student-content interactions through the systematic 
analysis of discussion forums and other forms of online communication. In this paper we use 
the QT model for analysing interactions in a virtual environment in higher education. This 
investigation extends previous research which found the model to be an effective tool with 
which to analyse the quality of assessment practice in the social sciences in the tertiary 
setting (Gore, Ladwig, Elsworth, Ellis, Parkes & Griffiths, 2009). 


Case Study: Mathematics Online 

Teaching mathematics online is a relatively new practice in which educators need to 
be aware not only of the affordances of the online medium, but also of issues inherent to 
mathematical concepts such as notation or the highly structured way in which concepts need 
to be scaffolded. These issues all play a major role in how courses are designed and delivered 
(Threlfall, Pool, & Homer, 2013). In pedagogical terms, Engelbrecht and Harding (2005) 
point out that while “little has been done in developing a pedagogy for online mathematics 
courses, there are some clear guidelines. Care should be taken to have a sound balance 
between teacher- and learner-centred activities and that interaction should be carefully 
planned; interaction between learner and content, between learner and instructor and between 
learner and learner” (p. 254). This is not specific to mathematics; in fact identical 
considerations are used for all learning areas with the proviso that evidence-based approaches 
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for improving online learning are used (Abrami, Bernard Bures, Borokhovski, & Tamim, 

2011 ). 

Our case study focused on the teaching of mathematics in the online environment to 
explore how a pedagogical model, the QT model, can be used for interpreting and evaluating 
teaching practices. Case studies are generally undertaken to ‘describe, explain or evaluate 
particular social phenomena’ (Gall, Gall & Borg, 2005, p.306) and in doing so, they aid in 
understanding complex social situations. In this study we use the lens of the QT model to 
systematically analyse the pedagogical features evident in two online courses. The two 
courses form part of a Master’s level program, intended for practising teachers and other 
educators who wish to gain postgraduate teaching qualifications in mathematics education. 
The student cohort we focused on for this study was comprised of 60 practising teachers who 
were re-training to be high school mathematics teachers. There were 34 female participants in 
the study (56.7%). The backgrounds of the students who enter this program are diverse, but 
most are secondary school teachers who have previously specialised in areas other than 
mathematics and believe that re-training in mathematics will provide better career prospects 
at a time when there is a shortage of mathematics teachers. 

We selected two concurrent semester-long courses in the Master’s program for the 
case study that had recently been updated with new technologies. Previously, these courses 
had been taught online asynchronously, whereby students were sent textbooks and other text- 
based course materials and asked to submit two written assignments and sit a final exam. In 
this earlier version, students were able to communicate via email with teaching academics to 
seek help with mathematical concepts or request feedback. They also participated in 
discussion board tasks in response to instructor prompts. In the revised offering of the courses 
we aimed to provide a wider range of online learning experiences for students. To do so, we 
utilised a range of digital resources available for online teaching and assessment to enable 
interactions of students to occur with each other and with the instructor in synchronous and/or 
asynchronous fashion (Holmes, 2005). 

The two courses were focused on mathematical concepts. The first (Course 1) focused 
on Calculus concepts including limits and continuity, derivatives and basic integration. The 
second (Course 2), contained elements of Number Theory, Combinatorics and a thorough 
introduction to Complex Numbers including their geometrical applications. Course 1 was 
undertaken by 41 students (46% female) and Course 2 by 50 students (62% female). There 
were 31 students who studied both courses (51% female). 

In general, the teaching of mathematics in online environments centres heavily on the 
mathematical concepts to be delivered. Using the constructs of the QT model (see Appendix 
1), the emphasis is customarily on the Intellectual Quality dimension of the teaching. In 
particular, the elements of Deep Knowledge, Deep Understanding, and Higher Order 
Thinking are often favoured. When we set out to improve the online delivery of the two 
courses in our case study, guided by the pedagogical principles underpinning the QT model, 
our primary focus was to also achieve a Quality Learning Environment, where Explicit 
Quality Criteria, Engagement, High Expectations, Social Support, Students’ Self-regulation 
and Student Direction would be more deliberate in our teaching. We also aimed to increase 
the Significance of the concepts we taught by including Narrative and Cultural Knowledge in 
our course design. 

To progress towards an improved Quality Learning Environment and increased 
Significance, we produced two types of resources during the intervention. On the one hand, 
and to facilitate Students’ Self-regulation, Engagement and Social Support, a series of 
resources were either specifically created for the course or externally sourced from the 
Internet. Externally sourced materials comprised two open-source text-based mathematics 
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books and many short videos and interactive demonstrations covering most of the topics in 
the course. Our internally produced materials included: 

• A course blog, which integrated many of the externally sourced short videos and 
interactive demonstrations. Our pedagogical aim with the course blog was to promote 
Engagement using a Narrative created by the lecturer and to include elements of the 
world history of mathematics thus promoting Cultural Knowledge. 

• Pencasts, i.e. interactive documents containing both written text and voice, were 
provided to students on request, thus promoting Student Direction, and frequently 
involved further explanations of mathematical problems raised in units of work, thus 
promoting Deep Understanding. 

• Discussion forums, aimed at promoting Social Support, were designed to enhance 
Intellectual Quality through discussion of the concepts in each of the units of work. 
The pencasts referred to above were included in these forums. 

In addition, to ensure Explicit Quality Criteria and High Expectations, we created a 
series of assessment tasks to be submitted fortnightly by made available to students from the 
beginning of the course. In previous iterations of these courses we had allowed students to 
submit handwritten mathematical work covering all topics at the end of the semester. This 
time we opted for two different forms of assessment to make better use of the online 
environment: a timed multiple-choice test to check for basic understanding of the topics 
presented in the preceding two weeks, and a more challenging long-response question to 
ensure Higher Order Thinking and Deep Understanding of the topics. The second task 
differed from previous years as it was to be typed, and plotted if necessary, with 
mathematical software provided to all students prior to the commencement of the course. 

This type of formative assessment was designed to help these teachers develop skills needed 
in modern-day technology-rich mathematics classrooms. The software used was suggested by 
practicing mathematics teachers who expressed how beneficial it would have been to learn to 
use it during their pre-service years. 


Methodology 

All data collection occurred in the first semester of 2013. Our case study comprises 
two separate courses, with different instructors, each involving six ‘units of work’. Each ‘unit 
of work’ was delivered fortnightly to students. Our university learning management system 
allows the running of analytics concerning use of the different resources included in the 
course, and we used these as the starting point for our analysis. In a previous paper (Prieto & 
Holmes, 2014) we presented a comprehensive analysis of the student activity and how it 
correlated with student achievement in the course, finding a positive relationship between the 
time spent within the learning management system and achievement in the course. In this 
previous paper, we used Engelbrecht and Harding’s (2005) framework to classify the 
interactions between students and academics in online mathematics courses. Their framework 
is comprised of seven factors ranging from instructor facilitation to internet resources, 
focusing mainly on the affordances of the online environment rather than on the pedagogical 
approach to teaching online. 

In this paper we analyse only the content of the forums where students discussed the 
units of work. By content we mean all written interactions occurring within the learning 
management system in the fortnightly period when that unit was delivered. As discussed in 
the introduction, content analysis of online interactions is often carried out by creating 
instruments specific to the online environment. However, in their review of content analysis 
schemes for discussion groups in online teaching, De Wever et al. (2006) expressed concern 
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about the instruments used. In particular they referred to the empirical validity of instruments 
and the reliability of coding. Our methodology addresses those concerns. By utilising the QT 
model, which has been systematically tested in classrooms, we use a coding instrument tested 
and validated in pedagogical research (Gore, 2007). The coding scales, used in relation with 
the focus question for each element as well as descriptions of each element/construct, are 
detailed in Appendix 2. 

To address issues of reliability in our study, the coding of content in the 273 posts 
contained in the discussion forums was undertaken by an experienced coder using the QT 
model. A random sample of 44 posts were independently doubled coded by another 
experienced coder, achieving an initial agreement of 85%. Subsequently, the two coders 
discussed the disparities and came to an agreed ‘best’ code for all posts. 

The forums were downloaded from our University’s learning management system by 
using its “Collect” functionality and then imported as text files into QSR NVivo 10 for 
coding and analysing purposes (NVivo qualitative data analysis Software; QSR International 
Pty Ltd. Version 10, 2012). For each unit of work the coding was conducted by highlighting 
portions of text in the discussion board corresponding to that unit, and coding it in relation to 
relevant elements of the QT model. The codes were pre-determined by creating “’Nodes’ 
within the NVivo environment. Double (or in some cases triple) coding was allowed since 
several elements could be present in the same portion of text. Examples of these extracts are 
given in the Results section. This approach enabled us to produce an accurate analysis not 
only of the degree to which the different elements in the QT model were evident in students’ 
interactions, but also the amount of text devoted to each of the elements as a percentage of 
the total amount of text students wrote in the forums. NVivo’s analytical tools enabled us to 
determine the amount of text that each of the different QT elements represented by using the 
Node summaries. It also enabled us to see the proportion of text coded to each element 
relative to the whole text for each forum examined. 

We only report here the interactions occurring in the forums that were not part of 
formal assessment for the courses. In other words, the focus of our study was on those online 
interactions that mimic the informal, but nevertheless crucial, connections between students 
and teachers which occur as part of typical face-to-face instruction. The discussion forums 
were organised with three different foci: assessment questions, mathematical questions, and 
miscellanea. The first forum was designed for students to communicate with their lecturer 
about all matters relating to assessment of the course, including mathematical questions 
which were part of their assignments. The second two forums were also monitored by the 
lecturer, but were used mainly by students to communicate amongst themselves, sharing 
resources and ideas or extending their learning beyond the course syllabus. All three forums 
have been analysed for this study. All individuals were de-identified for the purpose of this 
research. Human Research Ethics Committee approval at our institution was obtained 
(Approval No. H- 2013-0023), with active consent from students for us to anonymously 
report on their answers. 


Results 

In this section we provide results organised according to three different perspectives. 
First, we focus on the overall coding of each element in the QT model for each of the six 
units in both courses to get an overall picture of the pedagogical quality of the courses. Next, 
we analyse the proportion of text in the forums coded for each of the QT elements, and argue 
the limitations of this type text analysis for online forums. Lastly, we focus on in-depth 
content analysis of the text in the forums for each element of the QT model. 
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Overall QT Coding 

The agreed QT coding of the content in discussion forums for the six units in each of 
the two courses is presented in Figure 2. It is apparent that the scores given for each QT 
element were consistent across units demonstrating that certain features of the Quality 
Teaching model are more evident in these online courses than others. In particular, the 
elements of Deep Knowledge, Substantive Communication, Explicit Quality Criteria, Student 
Support, Student Self-Regulation and Inclusivity, scored highly in all units in both courses. In 
contrast, there was very little evidence found of Higher-order Thinking, Metalanguage, 
Cultural Knowledge and Narrative. In terms of the QT dimensions, Quality Learning 
Environment and Intellectual Quality consistently scored higher than Significance across all 
units of work. 

Course 1 
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Figure 2. Coding of all units of work in Course 1 and Course 2 
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The mean scores for each element in each module of each course are displayed in 
Figure 3. It is clear that although the courses were conducted independently by two different 
instructors, the overall pattern of scores is highly consistent across all 18 elements (r=0.96, 
n=18, p=.000). 


Proportion of Text 

Another measure of the prevalence of the QT elements across units is the percentage 
of text explicitly devoted to each element in the discussion boards. This measure could be 
considered equivalent to measuring the proportion of class time ‘devoted’ to each of the QT 
elements. 

The analysis was undertaken using text analysis software by highlighting portions of 
text and coding them against one or more QT elements as explained in the Methodology 
section. The software takes into consideration the total amount of text in each unit of work 
and consequently allocates a percentage to the text highlighted. Figure 2 presents the average 
coding over the 6 units of work for each of the elements in the course and the percentage of 
text devoted to each of the elements. 



-Q 

=5 

on 


Intellectual Quality Quality Learning Environment Significance 

Course 1 Course 2 • ■ Course 1 (%) #- Course 2 (%) 

Figure 3. Mean scores and proportion of text for each element 


The elements of Deep Knowledge, Substantive Communication, Explicit Quality 
Criteria, and Social Support are prevalent in both courses when analysed using the proportion 
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of text as a measure of their presence in the forums. This aligns with the overall coding 
presented in the previous section. Also similar to the overall coding, Deep Understanding, 
High Expectations and Knowledge Integration show medium levels of presence in the 
courses, and Metalanguage, Student Direction, Cultural Knowledge and Narrative are 
virtually non-existent in the discussion forums. 

One interesting point arising when comparing the two approaches to analysis of the 
courses, by overall coding and by proportion of text, is that some elements (i.e., Engagement, 
Student Self-regulation and Inclusivity), are coded higher in Table 1 than the percentage of 
text devoted to it in the coded discourse would seem to indicate at a first glance (see Figure 
3). For these elements, we considered that unsolicited student participation in the discussion 
board acted as a proxy measure of their presence. As an example of this, the element 
Engagement does not appear explicitly in any percentage of text in either of the forums, but 
we have coded it as high as 4 depending on the level and quality of unprompted engagement 
that students showed with the mathematical concepts taught that fortnight. If we were to 
continue the analogy with face-to-face settings, ‘proportion of text’ would be equivalent to 
‘time spent in class’ in a setting where students are highly engaged. Essentially, participation 
in the discussion forums implies engagement with the course, and so, all students who post 
comments are engaged to some degree. Therefore even though there is nothing in the text of 
student posts that indicates engagement, the existence of the text at all indicates the presence 
of this element. Taking this into consideration, we interpreted the presence of the comments 
as a proxy to engagement, and coded accordingly for the previous section. 

In the case of Student Self-Regulation, the fact that there is no text in the forums 
devoted to it is equivalent to having no time in a classroom when the teacher has to discipline 
students or redirect their attention to the task at hand. In this sense the absence of such text 
denotes high levels of self-regulation by students. Similarly, Inclusivity was not mentioned 
by the students but was considered evident in posts from diverse students including males and 
females. 


Analysis of Content 

Analysis of the content of interactions in relation to each of the QT elements will now 
be presented in turn, using examples where available to illustrate in detail how the 
characteristics of Quality Teaching can be observed in the online environment. 


Dimension: Intellectual Quality 

Deep Knowledge 

In both courses, across all units, the element of Deep Knowledge was coded highly 
indicating that within each unit the discussions focussed on a small number of key concepts 
and the relationships between those concepts. In many cases the discussion began with a 
student question: 

looking at the nature of points of inflection, a horizontal point of inflection 
occurs when both first and second derivitives [sic]= 0 and there is a change in 
concavity. An oblique point of inflection occurs when the first derivitive does 
not = 0 but there is a change in concavity and second derivative =0. Is this 
correct? 

Student post, Course 1, Unit 2 
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The question was typically followed by an instructor response: 

Points of inflection must all have their second derivative =0 (this is a necessary 
condition). However, the first derivative may or may not be 0. If the first 
derivative is NOT zero, they are sometimes referred to as "oblique". The 
function y=x A 3+x is an example of this. If the first derivative IS zero, then they 
are sometime referred to as "horizontal". The function y=x A 3 is an example of 
this. Please note that there must cdways be a change in concavity for it to be an 
inflection point. So answering your question: yes, you are correct:) 

Instructor post, Course 1, Unit 2 

In some cases, other students also offered responses to student queries, enlarging the 
discussion and thus demonstrating substantive communication as defined in the QT model. 
The units within each course were often accompanied by an instructor-written blog post (not 
included in the analysis conducted for this paper), introducing the concepts for the unit and 
thereby providing the necessary focus for the discussion that followed. These examples point 
to the necessity of planning cohesive, carefully focussed units within courses in order to 
promote discussions that go beyond superficial treatment of key concepts. 


Deep Understanding 

In comparison with the element of Deep Knowledge, the coding for Deep 
Understanding across both courses indicates that most students were demonstrating only 
superficial knowledge of the key concepts under discussion. This is to be expected because 
the forums were designed as spaces for the students to seek help. Instructors would be more 
likely to see this element demonstrated through the formal assessment tasks associated with 
each course, rather than in student posts. At times, however, the students did demonstrate 
Deep Understanding, particularly when helping each other: 

I think I can help with your question one query. 

If you included 1 as a value (by shading the circle) the equation would be 
undefined. This is because when you substitute x-1 back into the equation you 
would get 1-1=0 for the denominator. The denominator cannot be zero because 
anything divided by zero is undefined. 

I hope this makes sense. :) 

Student post, Course 1, Unit 6 

Problematic Knowledge 

Mathematics as a school subject is rarely presented as being socially constructed 
and/or open to question. It is not surprising that the transcripts analysed here show little 
evidence of the Problematic Knowledge element. There are several examples, however, 
where the instructors encourage students to embrace the fact that there are many different 
ways to complete most mathematics problems: 

Sometimes the graphs are long and skinny, you just have to tell me where the 
important features are. Sometimes you label them a,b,c etc or you could just 
state it under the graph. Sometimes we need a 1-1 scale, we don't want graphs to 
be stretched (we don't want a circle looking like an elipse). The question will be 
looking at your setting out, how do you communicate cdl the data to me in a neat 
and easy to read fashion. There may be different ways to do this, it is up to you 
how you set everything out and format your answer. 

Instructor post, Course 1, Unit 3 
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Higher-order Thinking 

There was little evidence of the Higher-order Thinking element in either course, 
however, this could be absent for similar reasons outlined above for Deep Understanding. 

The nature of online discussion boards to answer mathematical questions in the course 
encouraged brief, rather than extended interactions between participants, usually focussing on 
specific difficulties. Therefore, responses indicative of higher-order thinking about the course 
content, involving analysis, synthesis and evaluation, was not expected to be prevalent in 
these forums but expected in responses to formal assessment tasks which were not included 
in this analysis. 


Metalanguage 

While there was ample evidence of discussion using mathematical language, there 
was little evidence of discussion about mathematical language. One of the few examples of 
the Metalanguage element comes from an instructor post: 

Now into turning points: They ’re known as “turning points ” as this is where the 
curve ‘turns ’from ‘going up ’ to ‘going down ’ (or vice versa). Mathematically 
we can see this happening because the gradient changes from being positive to 
negative (or vice versa). 

Instructor post, Course 1, Unit 3 


Substantive Communication 

The Substantive Communication element is one of the most prevalent across all units 
in both courses. This is not surprising as any communication via an online discussion board, 
by necessity, has to be elaborated sufficiently for other participants to make sense of the post. 


Dimension: Quality Learning Environment 
Explicit Quality Criteria 

The Explicit Quality Criteria element refers to the extent to which the characteristics 
of high quality student work are made clear and are reinforced throughout the unit, ensuring 
that students are able to assess their own progress against these criteria. We found a 
consistently high level of evidence for this element across all units in both courses, typically 
in the form of students asking for detailed clarification of assessment task requirements or 
seeking feedback on their progress towards a high quality product. Also, students posted 
work-in-progress for feedback which was given by the instructor and in some cases by other 
students. 


Engagement 

The level of engagement of students in each course varied significantly across units, 
with some units having most students actively engaged in discussion and others with only a 
few engaged on a regular basis. In the online environment, engagement is on the one hand 
very easy to identify, as any posting indicates a conscious choice by the student to engage in 
conversation. However, student engagement could be occurring ‘behind the scenes’ with 
students accessing and reading posts but choosing not to engage by posting themselves. It is 
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difficult to determine the factors related to these different levels of engagement; this may be a 
fruitful area for future research. 


High Expectations 

This element refers to the degree to which students are engaged in challenging work 
and are encouraged to take risks. In general we found that only some students explicitly 
showed that they were engaging in challenging work, and indeed, it could be said that these 
students were taking a risk by posting their work publicly online as demonstrated by the 
following post: 

There is no solution for this question so I don 7 know if I got it correct. If anyone 
else has completed this question can you compare your answer to mine below, 
please. If you get something different can you put it up. If you get the same can 
you let me know too please. 

(u A 2-3v) A 4 = u A 8-12u A 6 v + 54u A 4v A 2-108u A 2 v A 3 +(S' / v A 4 Thanks 

(Student post, Course 2, Unit 5) 


Social Support 

The Social Support element is related to the degree to which the (online) learning 
environment is free from negative put downs and is an environment where contributions are 
valued and encouraged. Across all courses, no negativity was found; however, we generally 
observed a relatively neutral environment, with some positive and encouraging feedback such 
as the following comment from a student to the instructor: 

Thank you for providing this information and yes, 1 agree that a whiteboard 
would have been very handy. I have made notes from the video and I understand 
it better now. I really appreciate the time you take to answer our questions. 

Student post, Course 1, Unit 6 


And this from one student to another: 

Thank you for your brilliant explanations. I get it!! Yeah!! Thank you again and 
congratulations for working out lOd. I will need to read over it a couple more 
times before I get it but thanks for sharing. 

Student post, Course 2, Unit 5 


Students ’ Self-Regulation 

The element of Students’ Self-Regulation focusses on the extent to which students act 
autonomously when interacting in the online environment rather than only participating when 
prompted by others. Given the nature of the online learning environment, the element is 
coded highly across all units in both courses. Inherently, the online learning environment, 
particularly for adult learners, requires students to be self-regulating as they juggle their daily 
responsibilities with their learning trajectory, even when tight timelines are set for online 
activity. 
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Student Direction 

In the two courses under analysis there was very little opportunity for Student 
Direction, with students demonstrating only minimal amounts of control over the direction of 
their learning. These choices were mostly the pace at which they went over mathematical 
concepts and the assessment tasks, or alternative sources (videos, websites, etc.) they used for 
their learning. This has prompted the academic staff teaching the courses to re-design the 
structure of the discussion boards so that more input from students could be present. In 
particular the “Miscellanea” forum in both courses has been altered to include more Student 
Direction. We include an outline of those changes in the “Implications for course design” 
section below. 


Dimension: Significance 

Background Knowledge 

The presence of the element of Background Knowledge is measured through 
observation of references to students’ prior knowledge obtained within or outside formal 
educational experiences. Across all units in both courses there is little evidence of this 
element; however, occasionally students do share pertinent personal details: 

Thank you for sharing this article, I have an interest in this area. I am Food 
Tech trained and have always incorporated a lot of maths into my lessons. 1 
plan to do the reverse as a maths teacher. 

Student post, Course 1, Unit 4 


Cultural Knowledge 

The element of Cultural Knowledge is one of the lowest scoring elements across all 
units of work. In general, there was no acknowledgement made that alternative cultural 
approaches to mathematics are possible. This finding possibly reflects on the nature of 
mathematics as a discipline rather than on the online learning environment under analysis 
here. 


Knowledge Integration 

Knowledge Integration was found to be one of the most variable elements in the 
Significance dimension across all units. In one unit, several meaningful connections were 
evident between topic areas; however, in most of the units of work only trivial connections 
were made. For example, the following instructor post is illustrative of the Knowledge 
Integration element: 

Last week we dealt with some very important concepts when graphing 
functions: tangents and normals. This week we will build on those concepts, 
mash them up with what we know about continuity, intercepts and voila, we will 
be ready to graph any polynomial function that is thrown in front of us. Isn’t it 
great? 

Instructor post, Course 1, Unit 3 
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Inclusivity 

The element of Inclusivity is evident in the online environment through observation of 
the participation levels across units of work. In our analysis, most students contributed to the 
online discussion, although there was some level of variation in both courses. We did not find 
that the variation was gender-related, and other social differences were difficult to appreciate 
in an online environment. 


Connectedness 

Connectedness refers to the degree to which learning is related to ‘real world’ settings 
and/or students are given the opportunity to engage with an audience outside of the confines 
of the learning environment. In the case of online learning, the latter could possibly be 
achieved through engagement with the internet beyond the boundaries imposed by the 
learning management system, however, we found no evidence of this in either course. 


Narrative 

Across all units we found no evidence of the Narrative element. 


Implications for Course Design 

Implications for course design were drawn from applying the QT model to our case 
study of mathematics courses. First, the analysis of the 12 units of work from two online 
courses, with different instructors, produced remarkably consistent coding across the 18 
elements of the QT model which also aligns with what has been found previously in the face- 
to-face classroom-based environment (Gore, 2014). 

In terms of the Intellectual Quality dimension, we found that the elements of Deep 
Knowledge and Substantive Communication were most apparent in the online interactions 
across all units of work. While the other elements in this dimension were detectable, they 
were not as prevalent, possibly due to the concise nature of online discussions or alternatively 
due to the nature of mathematics as a subject. However, each of the lower coded elements, 
Deep Understanding, Problematic Knowledge, Higher-order Thinking and Metalanguage, 
could be enhanced in the forums through careful planning. Using the forums to pose higher- 
order tasks undertaken, as opposed to simply using these forums as a springboard for asking 
lecturers direct mathematical questions, could increase the quality of the courses. 

Considering the Quality Learning Environment dimension, the most highly coded 
element was Students’ Self-Regulation, however, in the online learning environment this 
element must be assumed to be present as adult students in higher education are, by 
definition, self-regulating. Secondly, the element of Explicit Quality Criteria was coded 
reasonably highly across all units, indicating students’ focus on explicit assessment 
requirements as a key component of learning. Three elements, Engagement, High 
Expectations and Social Support are all present in both courses, but do show some variation 
in coding across the units of work. The element of Student Direction is the lowest coded 
element in this dimension indicating a general lack of planned opportunities for students to 
influence the direction of their learning, with the exception of pacing. Opportunities to 
incorporate Student Direction needed to be provided in the courses, and subsequently new 
learning activities to achieve this goal have been incorporated in the Masters program. An 
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example of these activities has been adding a ‘question time’ to the weekly course blog and 
prompting students to ask about issues they would like their lecturer to expand on in the 
following blog post. This has effectively replaced the existing ‘Miscellanea” forum and 
added an extra component for Student Direction to it. Students who have taken the course 
since this change occurred have positively embraced this new feature and chosen topics 
relating not only to mathematical concepts but also current issues in mathematics education. 

Lastly we considered the dimension of Significance, which was consistently coded at 
a lower level than the other two dimensions. The highest coded element was Inclusivity 
which is observable in levels of participation among students from different social 
backgrounds. It should be noted that in our study, gender was the only obvious marker of 
social difference and we found no pattern in participation by gender. We found varying levels 
of the elements Student Background, Knowledge Integration and Connectedness, indicating 
that it is possible to observe these elements in the online learning environment. In contrast 
there was little or no evidence of the elements of Cultural Knowledge and Narrative in any of 
the units of work. 

The lack of certain Significance elements in the discussion forums may have had an 
impact on the students’ annual evaluation of courses undertaken by our institution. Using this 
official avenue, students provided feedback at the end of the course suggesting that they 
would have preferred the discussion forums and course blog to be blended into one interface. 
We interpret this feedback as an indication that Significance and the other two dimensions 
(Intellectual Quality and Quality Learning Environment) should have been integrated to 
provide a more cohesive learning experience for students, instead of being delivered using 
different media: the forums and the blog. It is important to remember here that both courses 
included a weekly blog delivered to students independently of the student forums. The 
weekly course blogs were purposely created to incorporate Narrative and Cultural 
Knowledge into the courses but since they didn’t include student interactions, they are not 
part of the analysis reported in this paper. 

Our exploration of the courses’ pedagogy using the QT model, through a coding 
process that gives specificity and structure to key points of consideration, reveals some 
consistent areas of strength in the methods employed in the online teaching of mathematics, 
specifically a clear focus on Deep Knowledge and Explicit Quality Criteria. Also, the nature 
of the virtual environment, where students are only observable based on their online postings 
and interactions, ensures that Substantive Communication, Students’ Self-regulation, 
Inclusivity and Social Support, are consistently displayed. In contrast, evidence to support the 
presence of the remaining elements was variable across units of work and clearly exposed 
areas ripe for pedagogical improvement. Interestingly, some of these areas of improvement 
were independently confirmed by student feedback upon completion of the courses. The main 
such area identified was a need for a thorough integration of the course blog and the student 
forums. 


Conclusions 

With this paper we demonstrate the applicability of evidence-based methods for 
appraising quality teaching in face-to-face classrooms to quality teaching in online 
environments. In particular, we illustrate how the Quality Teaching model can be used to 
analyse, review and improve teaching in the online environment using a case study of a series 
of mathematics courses for practising teachers who wish to gain postgraduate qualifications. 
While we acknowledge that our case study is small in scope and results are not generalizable, 


Vol 41, 8, August 2016 


36 



Australian Journal of Teacher Education 


we contend that the Quality Teaching model can be a useful analytical tool for improving 
online learning environments. 

We utilised a range of digital resources available for online teaching focusing on 
creating a Quality Learning Environment, where the work of students is Significant and of 
high Intellectual Quality. This type of environment is explained in detail in the Quality 
Teaching model (NSW Department of Education and Training, 2003a). Our research 
indicates that an evidence-based pedagogical model is a feasible model for analysing, 
understanding and improving online teaching. The model clearly identified some strengths 
within our current practices but also revealed some important areas for improvement, in 
particular in the dimension of Significance. 

Online instruction is often guided by the affordances of technological tools; however 
it is clear that such a limited focus can omit vital components of quality teaching. Our study 
supports the view expressed by Margaryan, Bianco and Littlejohn (2015), that general 
pedagogical principles should be explored and applied to any online learning environment 
regardless of size and scope. We contend that a classroom-based pedagogical framework, the 
Quality Teaching model, can be a comprehensive tool for directing pedagogical improvement 
in online learning. This kind of analysis can assist online learning instructors to supplement 
their instructional strategies with consideration of key characteristics of quality face-to-face 
teaching which can be overlooked in the virtual environment. 
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